Stream generation apparatus, stream generation method, coding apparatus, coding method, recording medium and program thereof

ABSTRACT

A stream generation apparatus generates a stream including coded pictures and a command for managing a buffer which holds a decoded picture, the command being added to one of the coded pictures as a reference picture. A judging unit judges whether or not the coded picture to which the command is added is skipped at the time of trick-play, and an adding unit adds, in the case where the coded picture is judged to be skipped, repetition information indicating the same contents as the command to another coded picture that follows, in decoding order, the coded picture judged to be skipped and that is not skipped at the time of the trick-play. A generating unit generates the stream including the coded pictures, the command and the repetition information.

This application is a continuation of U.S. application Ser. No.10/580,468, filed May 25, 2006 now U.S. Pat. No. 7,889,788, which is thenational stage of PCT/JP2005/008318, filed Apr. 25, 2005, the contentsof which is fully incorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Technical Field

The present invention relates to a stream generation apparatus whichgenerates a stream including coded pictures, a stream generation method,a picture coding apparatus, a picture coding method, a recording mediumand a program thereof.

2. Background Art

In the age of multimedia which integrally handles audio, video and otherpixel values, existing information media, specifically, newspaper,magazine, television, radio, telephone and the like through whichinformation is conveyed to people, have recently come to be included inthe scope of multimedia. Generally, multimedia refers to something thatis represented by associating not only characters, but also graphics,sound, and especially images and the like, together, but in order toinclude the aforementioned existing information media in the scope ofmultimedia, it becomes a prerequisite to represent such information in adigital form.

However, if the amount of information carried by each of the mentionedinformation media is estimated as the amount of digital information,while the amount of information for 1 character in the case of text is 1to 2 bytes, the amount of information required for sound is 64 Kbits persecond (telephone quality), and 100 Mbits or over per second becomesnecessary for moving pictures (current television receiving quality), itis not realistic for the information media to handle such an enormousamount of information as it is in digital form. For example, althoughvideo phones are already in the actual use via Integrated ServicesDigital Network (ISDN) which offers a transmission speed of 64 Kbit/s to1.5 Mbit/s, it is impossible to transmit images on televisions andimages taken by cameras directly through ISDN.

Accordingly, information compression techniques have become required,and for example, in the case of the video phone, the H.261 and H.263standards for moving picture compression technology, internationallystandardized by the International TelecommunicationUnion—Telecommunication Standardization Sector (ITU-T), are beingemployed. Moreover, with MPEG-1 standard information compressiontechniques, it has also become possible to store video information ontogeneral music compact discs (CD) together with audio information.

Here, Moving Picture Experts Group (MPEG) is an international standardfor moving picture signal compression standardized by InternationalOrganization for Standardization and International ElectrotechnicalCommission (ISO/IEC). The MPEG-1 is a standard for compressing movingpicture signals up to 1.5 Mbps, in other words, compressing televisionsignals up to approximately a hundredth part. Moreover, since targetpicture quality within the scope of the MPEG-1 standard is limited to amedium degree of quality which can be realized by a transmission speedof primarily about 1.5 Mbps, the use of MPEG-2, which was standardizedto satisfy demands for further improved picture quality, realizestelevision broadcasting quality with moving picture signals compressedto 2 to 15 Mbps. Furthermore, currently, MPEG-4, which has exceededMPEG-1 and MPEG-2 compression ratios, and also enables coding, decodingand operating on a per-object base, and realizes the new functionsrequired for the multimedia age, has been standardized by the work group(ISO/IEC JTC1/SC29/WG11) that has promoted the standardization of MPEG-1and MPEG-2. The MPEG-4 was initially aimed at standardizing a low bitrate coding method. However, currently, this has been expanded to thestandardization of a more versatile coding method further including highbit rate coding for interlaced pictures. After that, MPEG-4 AdvancedVideo Coding (AVC) is standardized as a next generation picture codingmethod with higher compression ratio by a cooperation of ISO/IEC andITU-T. It is prospected to be used for next generation optical discrelated devices or for a broadcast directed to cell phone terminals.

Generally, in coding of a moving picture, the amount of information iscompressed by reducing redundancy in temporal and spatial directions.Accordingly, in an inter-picture prediction coding which aims atreducing the temporal redundancy, a motion estimation and a generationof a predictive picture are performed on a block-by-block basis byreferring to a preceding or following picture, and a difference valuebetween the obtained predictive picture and a picture to be coded iscoded. Here, a picture indicates a screen: it indicates a frame in aprogressive picture; and it indicates a frame or a field in aninterlaced picture. Here, the interlaced picture is a picture whoseframe is made up of two fields which differ temporally each other. In acoding and decoding of the interlaced picture, it is allowed to processone frame as a frame, to process it as two fields, or to process it as aframe structure or as a field structure on a block-by-block basis in theframe.

An I picture is a picture that is intra coded without referring to areference picture. Also, a P picture is a picture that is inter-pictureprediction coded by only referring to one picture. Further, a B pictureis a picture that can be inter-picture prediction coded by referring totwo pictures at the same time. The B picture can refer to two picturesas a pair of any pictures which are displayed before or after the Bpicture. A reference picture can be specified for each block which is abasic unit for coding and decoding. The reference picture which isprecedently described in a coded bit stream is distinguished as a firstreference picture with the reference picture which is subsequentlydescribed as a second reference picture. Note that, as a condition forcoding and decoding these pictures, it is necessary that a picture to bereferred has already been coded and decoded.

FIG. 1 is a drawing showing a structure of a stream of the conventionalMPEG-2. As shown in FIG. 1, the stream of the MPEG-2 has a hierarchicalstructure as described in the following. The stream is made up of morethan one Groups of Pictures (GOP), and an editing and random accessingof a moving picture are allowed by using the stream as a basic unit forcoding. Each GOP is made up of more than one picture. Each picture isone of an I picture, a P picture or a B picture. Each stream, GOP andpicture is further made up of synchronous code (sync) which indicates abreakpoint of each unit and a header which is common data in the unit.

FIG. 2A and FIG. 2B are drawings showing an example of a predictivestructure among pictures used in MPEG-2. In the drawings, pictures shownas diagonally shaded area are pictures to be referred by other pictures.As shown in FIG. 2A, in MPEG-2, P picture (P0, P6, P9, P12, P15) can beprediction coded by referring to an I picture or P picture that isdisplayed immediately before the P picture. Further, B picture (B1, B2,B4, B5, B7, B8, B10, B11, B13, B14, B16, B17, B19, B20) can beprediction coded by referring to an I picture or P picture that isdisplayed prior to and following to the B picture Furthermore, arrangingorder in a stream has been determined as follows: the I pictures and Ppictures are arranged in displaying order; and each of the B pictures isarranged immediately after an I picture or P picture that is displayedimmediately after the B picture. As a GOP structure, for example, asshown in FIG. 2B, pictures from I3 to B14 can be included in one GOP.

FIG. 3 is a drawing showing a structure of a stream of MPEG-4 AVC. InMPEG-4 AVC, there is no concept equivalent to the GOP. Therefore, in thecase where an arrangement method of parameter sets that are describedlater and predictive structure of pictures are not constrained, it isnecessary to search a picture whose picture data is sequentiallyanalyzed and can be decoded when randomly accessed. However, byseparating data into special picture units by which each picture isdecoded without depending on other pictures, it is possible to constructa unit which can be randomly accessed and is equivalent to the GOP. Suchseparated units are called random access units (RAU) and a stream whichis made up of RAUs is called a stream having a random access structure.

Here, it is explained about the access unit (hereafter referred to asAU) which is a basic unit for dealing with a stream. An AU is a unitused for storing coded data in one picture, including parameter sets andslice data. The parameter sets are divided into a picture parameter set(PPS) which is data corresponding to a header of each picture and asequence parameter set (SPS) which is corresponding to a header of aunit of GOP in MPEG-2 and more. The SPS includes a maximum number ofpictures available for reference, picture size and the like. The PPSincludes a variable length coding method, an initial value ofquantization step, and a number of reference pictures. An identifierindicating which one of the PPS and SPS is referred is attached to eachpicture.

For the I pictures of MPEG-4 AVC, there are two types of the I pictures:an Instantaneous Decoder Refresh (IDR) picture; and an I picture whichis not the IDR picture. The IDR picture is an I picture which can bedecoded without referring to a picture preceding to the IDR picture indecoding order, that is, whose condition necessary for decoding isreset, and is equivalent to a leading I picture of a closed GOP ofMPEG-2. For the I picture which is not the IDR picture, a picture whichfollows the I picture in decoding order may refer to a picture which ispreceding to the I picture in decoding order. Here, the IDR picture andI picture indicate pictures made up of only I slices. The P pictureindicates a picture made up of P slices or I slices. The B pictureindicates a picture made up of B slices, P slices or I slices. Note thatthe slices of the IDR picture and the slices of the non-IDR picture arestored in different types of NAL units.

The AU of MPEG-4 AVC can include, in addition to data necessary fordecoding a picture, supplemental information called SupplementalEnhancement Information (SEI) which is unnecessary for decoding apicture, boundary information of AU and the like. The data such asparameter set, slice data and SEI are all stored in a NetworkAbstraction Layer (NAL) unit (NALU). The NAL unit is made up of a headerand a payload, and the header includes a field which indicates a type ofdata stored in the payload (hereafter referred to as NAL unit type). Thevalue of the NAL unit type is defined for each type of data such as aslice and SEI. By referring to the NAL unit type, the type of datastored in the NAL unit can be specified. The NAL unit of SEI can storeone or more SEI messages. The SEI message is also made up of a headerand a payload and a type of information stored in the payload isidentified by a type of SEI message indicated in the header.

FIG. 4 is a drawing showing an example of a predictive structure of theMPEG-4 AVC. In MPEG-4 AVC, an AU of P picture can refer to an AU of Bpicture. As shown in FIG. 4, the AU of P picture (P7) can refer to theAU of B picture (B2). Herein, in order to perform high-speed playback bydisplaying only the AUs of I pictures and P pictures, I0, B2, P4 and P7have to be decoded. Thus, when trick-play such as jump-in playback,variable-speed playback or reverse playback is performed, the AUsnecessary to be decoded cannot be determined in advance so that all AUsneed to be decoded in the end. However, by storing, in a stream,supplemental information indicating AUs necessary to be decoded for thetrick-play, the AUs to be decoded by referring to the supplementalinformation can be determined. Such supplemental information is calledtrick-play information. Further, if a constrain is previously set in apredictive structure such as that the AUs of P pictures do not refer toan AU of B picture, only the AUs of the I pictures and P pictures can bedecoded and displayed. Furthermore, for the AUs of I pictures and Ppictures, the AUs of I pictures and P pictures can be sequentiallydecoded and displayed if the decoding order is same as the displayingorder.

FIG. 5 is a block diagram showing a structure of a conventionalmultiplexer.

A multiplexer 17 is a multiplexer which receives a video data, codes theinputted video data into streams of MPEG-4 AVC, generates databaseinformation about the coded data, multiplexes and records the coded dataand the database information. It includes a stream attributedetermination unit 11, a coding unit 12, a database informationgeneration unit 13 having a general database information generation unit14, a multiplexing unit 15 and a recording unit 16.

The stream attribute determination unit 11 determines a coding parameterfor coding the MPEG-4 AVC and a constrained matter relating to atrick-play, and outputs them to the coding unit 12 as attributeinformation TYPE. Here, the constrained matter relating to thetrick-play includes information about whether or not to apply aconstraint for constructing a random access unit to a stream of theMPEG-4 AVC, whether or not to include information indicating an AU to bedecoded when variable speed playback or reverse playback is performed,or whether or not to give a constrain on a predictive structure amongAUs. The coding unit 12, based on the attribute information TYPE, codesthe inputted video data into a stream of the MPEG-4 AVC, and outputs theaccess information in the stream to a general database informationgeneration unit 14 while outputting the coded data to the multiplexingunit 15. Here, the access information indicates information on an accessbasis which is a basic unit for accessing to a stream, including a startaddress, size, displayed time and the like of a leading AU in the accessbasis. The stream attribute determination unit 11 further outputsinformation necessary for generating database information such as acompression method and a resolution as general database information tothe general database information generation unit 14. The databaseinformation generation unit 13 generates database information, and ismade up solely of the general database information generation unit 14.The general database information generation unit 14 generates, with theaccess information and the general database information, a table data tobe referred when accessing to a stream and a table data in whichattribute information such as a compression method are stored, andoutputs the generated table data to the multiplexing unit 15 as databaseinformation INFO. The multiplexing unit 15 generates multiplexed data bymultiplexing the coded data and the database information INFO, andoutputs the multiplexed data to the recording unit 16. The recordingunit 16 records the multiplexed data inputted from the multiplexing unit15 into an optical disc, a hard disc or a recording medium such as amemory.

FIG. 6 is a block diagram showing a structure of a conventionaldemultiplexer.

A demultiplexer 27 is a demultiplexer which, in accordance with anexternally inputted command which instructs to perform trick-play,separates, decodes and displays the AU data of MPEG-4 AVC from theoptical disc on which a stream of the MPEG-4 AVC is recorded togetherwith the database information. It includes a database informationanalyzing unit 21, a decoding/displaying AU determination unit 23, an AUseparation unit 24, a decoding unit 25, and a displaying unit 26.

The database information analyzing unit 21 is made up solely of thegeneral database information analyzing unit 22. A trick-play instructionsignal for instructing to perform trick-play such as variable speedplayback, reverse playback or jump-in playback is inputted to thegeneral database information analyzing unit 22. When the trick-playinstruction signal is inputted, the general database informationanalyzing unit 22 analyzes the inputted signal by obtaining accessinformation ACS from the database information of the multiplexed data,obtains access destination information including address information ofan access basis in which an AU which is to be decoded or displayed isincluded and the like, and notifies the AU separation unit 24. The AUseparation unit 24 analyzes AUs which make up an access basis, obtainsthe trick-play information TRK about an AU to be decoded and displayed,and outputs the obtained information to the decoding/displaying AUdetermination unit. The decoding/displaying AU determination unitdetermines an AU to be decoded and displayed based on a predeterminedrule, and notifies the identification information of the AU to bedecoded and the identification information of the AU to be displayedrespectively to the AU separation unit 24 and the displaying unit 26.The AU separation unit 24 separates the data in the AU to be decodedbased on the access destination information, and outputs the separateddata to the decoding unit 25. The decoding unit 25 decodes the inputtedAU data, and outputs the decoded data to the displaying unit 25.Finally, the displaying unit 26 selects an AU which is indicated to bedisplayed in the display AU information, and displays the selected AU.(Refer to Japanese Laid-Open Patent Publication No. 2003-18549).

SUMMARY OF INVENTION

In the decoding apparatus, reference pictures or pictures which arewaiting to be displayed are stored in a buffer memory called DecodedPicture Buffer (DPB) after the pictures are decoded. However, predictivestructure of a picture is flexible in MPEG-4 AVC so that memorymanagement in the DPB becomes complicated. Thus, there is a problem thatit is difficult to perform trick-play such as fast forward. For example,in the case where high-speed playback which only decodes and displays Ipictures and P pictures is performed, memory management information maybe stored in a B picture to be skipped. If the memory managementinformation indicates to hold a specific picture in a DPB withoutdeleting it from the DPB, the information cannot be obtained. Therefore,the specific picture may be deleted from the DPB and the following Ipicture or P picture which refer to the specific picture may not be ableto be decoded.

FIG. 7A and FIG. 7B are diagrams showing memory management of the DPB.As shown in FIG. 7A, picture data for a plurality of frames can bestored in the DPB. In this example, picture data for four frames can bestored. Also, in the DPB, two types of areas can be set: a long termmemory; and a short term memory. The picture data stored in short termmemories are bumped out sequentially from a picture of the earliestdecoding order. On the other hand, the picture data stored in a longterm memory is held in a DPB and can be referred by other pictures untilit is set so as not to be referred by other pictures, by a memorymanagement command called a Memory Management Control Operation (MMCO).For example, a long term memory is used when it is necessary to storethe I picture for a longer term, such as when a leading I picture in therandom access basis is referred by following pictures in decoding order.Note that the pictures stored in the short term memories also can bemade non-referenced by the MMCO. For default, each picture is stored ina short term memory. Here, the memory management command can specifyabout how many memories for how many frames are allocated respectivelyfor the long term memory and the short term memory. Note that the memorymanagement command can be issued solely to the reference pictures. FIG.7B shows an example where a memory for one frame is allocated for along-term memory and memories for three frames are allocated for a shortterm memory out of frame memory for four frames. The allocation of thelong term memory and the short term memory can be changed dynamicallydepending on the memory management command MMCO.

FIG. 8 shows an example of a use of the memory management command. FIG.8( a) shows an arrangement of pictures in a random access basis. In thedrawing, I, P, B and Br are respectively indicate an I picture, a Ppicture, an non-referenced B picture, and a reference B picture. Thenumber attached to each picture indicates a displaying order. Here, thenon-referenced picture B indicates a B picture which is not referred byother pictures, and the reference B picture indicates a B picturereferred by other pictures. Also, arrows indicate predictive structures.For example, P9 indicates that it refers to P5 and I1, B2 refers to I1and Br3, and Br3 refers to I1 and P5. The P pictures only refer to the Ipictures or the P pictures so that the reference B picture is notreferred. FIG. 8( b) shows pictures shown in FIG. 8( a), arranged indecoding order. Here, in Br11, it is assumed that a memory managementcommand for transferring I1 to a long term memory is stored in headerinformation of slices which constitute a picture. FIG. 8( c) to (h)shows pictures stored in a DPB when the DPB can store picture data forfour frames. Here, Br refers to only an I picture or a P pictureimmediately before or after the Br in displaying order, and the Br isdeleted from DPB according to the memory management command of an Ipicture or P picture which are two pictures after the Br in thedisplaying order. FIG. 8( c) shows pictures stored in the DPB after Br3is deleted at P9. In the DPB, I1, P5 and P9 are all stored in the shortterm memory. After P13 is decoded, as shown in FIG. 8( d), I1, P5, P9and P13 are stored in the DPB. Herein, DPB is full because four picturesare stored. Following that, after the Br11 is decoded, the Br11 shouldbe stored in the DPB. However, since the DPB is full, it is necessary todelete one of the pictures stored in the DPB. Here, originally the I1which was decoded at the earliest should be removed from the DPB.However, it is shown that a long term memory is assigned for the I1 soas to transfer the I1 to the long term memory (FIG. 8( e)). Accordingly,when Br11 is stored, as shown in FIG. 8( f), P5 which is decoded at theearliest is deleted from the pictures stored in the short term memories.Note that, the deletion of P5 is also executed by the memory managementcommand so that the memory management command stored in the Br11includes a command which specifically indicates to make the P5non-referenced, in other words, indicates that the P5 can be deleted.FIG. 8( g) shows a state when Br11 is deleted and P17 is stored in theDPB after P17 is decoded. Lastly, while P21 refers to the I1, the I1 istransferred to a long term memory when P21 is decoded and available forreference so that the I1 can be referred without any problems (FIG. 8(h)).

Next, it is explained about a problem of memory management whentrick-play such as fast forward and jump-in playback is performed.Especially, a fast-forward playback by decoding and playing back only Ipictures and P pictures (IP playback) has been introduced for a generaluse. FIG. 9 shows memory management when the random access unit that issame as what is shown in FIG. 8, is IP-played back. First, afterdecoding I1, P5 and P9, the I1, P5 and P9 are stored in short termmemories of DPB as shown in FIG. 9( c). Next, after P13 is stored, fourof the I1, P5, P9 and P13 are stored in the DPB so that the DPB becomesfull herein (FIG. 9( d)). After that, originally, Br11 is set for a longterm memory for storing I1 depending on a memory management command andthe I1 is transferred to a long term memory. However, the Br11 isskipped without being decoded so that I1 remains to be stored in theshort term memory. Accordingly, when decoded P17 is stored in DPB, theI1 is deleted because it has the earliest decoding order among picturesstored in the short term memories (FIG. 9( e)). Therefore, there arefour pictures of P5, P9, P13 and P17 are stored in the DPB when P21 isdecoded so that P21 cannot be decoded because there is no I1 (FIG. 9(f)). Thus, in the case where a picture is decoded while skippingpictures when trick-play is performed, if a picture in which a memorymanagement command is stored is skipped, it causes a problem of abreakdown of memory management so that following pictures cannot bedecoded correctly. Therefore, the IP playback cannot be realized withoutdecoding all reference pictures and the amount of processing accordingto the IP playback has been increased.

An object of the present invention is to provide a stream generationapparatus, a stream generation method, a picture coding apparatus, apicture coding method, a recording medium and a program which, whentrick-play is performed, prevents a breakdown of trick-play due to thereason that there is no reference picture necessary for decoding in abuffer, and realizes easily the trick-play of a picture by skipplayback.

In order to achieve the above object, a stream generation apparatusaccording to the present invention is the stream generation apparatuswhich generates a stream including coded pictures and a command formanaging a buffer which holds a decoded picture as a reference picture,the command being added to one of the coded pictures, the apparatuscomprising: a judging unit operable to judge that whether or not thecoded picture to which the command is added is to be skipped at the timeof trick-play; an adding unit operable to add, in the case where thecoded picture is judged to be skipped, repetition information indicatingthe same contents as the command to another coded picture that follows,in decoding order, the coded picture judged to be skipped and that isnot skipped at the time of the trick-play; and a generating unitoperable to generate the stream including the coded pictures, thecommand and the repetition information.

According to this structure, a breakdown of trick-play, which is causedby a reason that there is no reference picture necessary for decoding isin a buffer, can be prevented. In other words, the trick-play can beeasily realized by skip playback of pictures.

Here, the command may instruct to change an attribute of the referencepicture stored in the buffer from a short term memory to a long termmemory.

According to this structure, the trick-play can be easily realized evenin the case where there are two types of short term and long termreference pictures or in the case where the relationship among picturesis complicated.

Here, the judging unit may be operable to judge that a reference Bpicture is skipped at the time of trick-play, in the case where thecoded picture to which the command is added is the reference B picturethat is to be referred to when another coded picture is decoded.

In addition, the adding unit may be further operable to add therepetition information to one of an I picture and a P picture thatfollows, in decoding order, the reference B picture judged to beskipped.

According to this structure, in the case where the B pictures are alsoused as reference pictures in addition to the I pictures and P pictures,necessary reference pictures can be stored in a buffer for sure evenwhen only I pictures and P pictures are skip-played back.

Here, the judging unit is operable to judge that a P picture is skippedat the time of trick-play, in the case where the coded picture to whichthe command is added is the P picture that is to be skipped when aspecific P picture is decoded, and the specific P picture can be decodedby selectively decoding a preceding I picture or P picture in decodingorder.

In addition, the adding unit may be operable to add the repetitioninformation to the another picture that follows, in decoding order, theP picture judged to be skipped and that is necessary for decoding thespecific P picture.

According to this structure, in the case where a skip playback using theaccess point P picture is performed, necessary reference pictures can bestored in a buffer for sure.

Further, the same units are included in the present invention of astream generation method, a picture coding apparatus, a picture codingmethod, a recording medium and a program.

Further Information about Technical Background to this Application

The disclosures of the following Japanese Patent Applications includingspecification, drawings and claims are incorporated herein by referenceson their entirety: Japanese Patent Application No. 2004-134211 filed onApr. 28, 2004 and Japanese Patent Application No. 2004-272517 filed onSep. 17, 2004.

BRIEF DESCRIPTION OF DRAWINGS

These and other objects, advantages and features of the invention willbecome apparent from the following description thereof taken inconjunction with the accompanying drawings that illustrate a specificembodiment of the invention. In the Drawings:

FIG. 1 is a drawing showing a stream structure in a MPEG-2 video.

FIG. 2A and FIG. 2B are drawings showing an example of a GOP structurein the MPEG-2 video.

FIG. 3 is a drawing showing a stream structure of a MPEG-4 AVC.

FIG. 4 is a drawing showing an example of a predictive structure of theMPEG-4 AVC.

FIG. 5 is a block diagram showing a structure of a conventionalmultiplexer which codes a stream of the MPEG-4 AVC and multiplexes thecoded stream.

FIG. 6 is a block diagram showing a structure of a conventionaldemultiplexing apparatus which plays back multiplexed data generated bythe conventional multiplexer.

FIG. 7A and FIG. 7B are drawings indicating memory management in adecoded picture/buffer in the MPEG-4 AVC.

FIG. 8 is a drawing showing an example when a use of a memory managementcommand is necessary.

FIG. 9 is a drawing showing memory management when I and P pictures in arandom access unit that is shown in FIG. 9 are played back.

FIG. 10 is a block diagram showing a structure of a coding apparatusaccording to a first embodiment.

FIG. 11 is a drawing showing a method of repeating a memory managementcommand.

FIG. 12A is a drawing showing pictures and a memory management commandwhen AP-P picture is used in the conventional technology.

FIG. 12B is a drawing showing pictures and a memory management commandwhen AP-P picture is used according to the first embodiment.

FIG. 13 is a flowchart showing a coding method for realizing memorymanagement without causing a breakdown when I and P pictures are playedback.

FIG. 14 is a flowchart showing a coding method for realizing memorymanagement without causing a breakdown when AP-P picture is decoded.

FIG. 15 is a block diagram showing a decoding apparatus which realizes adecoding method according to a sixth embodiment.

FIG. 16 is a flowchart showing a method of decoding a coded stream whichis guaranteed that memory management without causing a breakdown can berealized when I and P pictures are played back.

FIG. 17 is a flowchart showing a method of decoding a coded stream whichis guaranteed that memory management without causing a breakdown can berealized when an AP-P picture is played back.

FIG. 18 is a block diagram showing a structure of a first multiplexeraccording to a second embodiment.

FIG. 19A and FIG. 19B are diagrams showing contents of playback supportinformation.

FIG. 20 is a drawing showing a method for specifying a NAL unit in whichthe playback support information is stored.

FIG. 21 is a flowchart showing an operation of the first multiplexer.

FIG. 22 is a block diagram showing a structure of a second multiplexeraccording to a third embodiment.

FIG. 23 is a block diagram showing a demultiplexer according to a fourthembodiment.

FIG. 24 is a flowchart showing a first operation of the demultiplexer.

FIG. 25 is a flowchart showing a second operation of the demultiplexer.

FIG. 26 is a diagram showing data hierarchy of HD-DVD according to afifth embodiment.

FIG. 27 is a diagram showing a structure of logical space on the HD-DVD.

FIG. 28 is a diagram showing structure of a VOB information file.

FIG. 29 is explanatory drawing of a time map.

FIG. 30 is a diagram showing a play list file.

FIG. 31 is a diagram showing a structure of a program file correspondingto the play list.

FIG. 32 is a diagram showing a structure of a BD disc total databaseinformation file.

FIG. 33 is a diagram showing a structure of a file for recording globalevent handler.

FIG. 34 is a schematic block diagram showing the HD-DVD player accordingto the sixth embodiment.

FIG. 35A, FIG. 35B and FIG. 35C show recording media in which a programfor realizing a picture coding method and a picture decoding method ofthe present invention.

DETAILED DESCRIPTION OF THE INVENTION

Hereafter, it is explained about embodiments of the present inventionwith references to drawings.

First Embodiment

In this embodiment, it is explained about a coding apparatus and adecoding apparatus which can obtain a command necessary for managingmemories in a DPB only from pictures necessary for skip playback whentrick-play is performed.

The coding apparatus generates a stream including a memory managementcommand and coded pictures. When the stream is generated, the codingapparatus judges whether or not a coded picture to which a memorymanagement command is added is skipped when trick-play is performed, inthe case where it is judged as a coded picture to be skipped, repetitioninformation indicating the same contents as the command is added to acoded picture which is not skipped and is decoded after the codedpicture to be skipped, when trick-play is performed.

FIG. 10 is a block diagram showing a structure of a coding apparatus1000 in the present embodiment. The coding apparatus 1000 includes apicture type determination unit 1001, a repetition judgement unit 1002,a repetition information generation unit 1003, a picture coding unit1004, and a coded data output unit 1005. The picture type determinationunit 1001 determines a picture type of a picture to be coded and inputsthe determined picture type Pt to the repetition judgement unit 1002 andthe picture coding unit 1004. The picture coding unit 1004 codes aninputted picture Vin depending on the picture type Pt, inputs the codeddata pic to the coded data output unit 1005, and inputs memorymanagement information mmco to the repetition judgement unit 1002. Ifthe memory management information mmco is not issued for the codedpicture, that is indicated in the memory management information mmco.The repetition judgement unit 1002 judges whether or not to repeat amemory management command based on the memory management informationmmco and the picture type Pt, and inputs a result of the judgment to therepetition information generation unit 1003 as a repetition command Re.The repetition information generation unit 1003 generates a DRPMR SEIwhen the repetition command Re instructs to repeat the memory managementcommand, and inputs the SEI data sei to the coded data output unit 1005.Here, when the repetition command Re instructs to repeat the memorymanagement command, information necessary for generating the DRPMR SEIis also inputted to the repetition information generation unit 1003. Thecoded data output unit 1005 outputs the coded data pic and SEI data sei.

Thus, the SEI data sei generated according to the repetition command Reincludes same contents as the memory management information mmco, andsubstantially includes a copy of the memory management information mmco.

FIG. 11 shows a random access unit of a MPEG-4 AVC stream stored in theinformation recording medium as an example of a stream coded by thecoding apparatus 1000 in the present embodiment. While the example issame as the conventional example shown in FIG. 9, it differs with theconventional example in that a memory management command stored in Br11is repeated at P17 using Decoded Reference Picture Marking RepetitionSupplemental Enhancement Information (hereafter referred to as DRPMRSEI). Specifically, a memory management command set in Br11 fortransferring I1 to a long term memory by making P5 non-referenced isrepeated in P17. Accordingly, even if Br11 is skipped when the IP isplayed back, it is known that the I1 is transferred to a long termmemory at Br11 when P17 is decoded. Consequently, I1 is transferred tothe long term memory, P5 is deleted from the DPB after P17 is decoded,and P17 is stored instead (FIG. 11( e)). Therefore, as shown in FIG. 11(f); there is I1 in DPB when P21 is decoded so that P21 can be decoded byreferring to I1. Thus, when a memory management command is issued to areference B picture, I pictures and P pictures can be decoded withoutbreaking down the memory management even when IP playback is performed,by repeating the memory management command using DRPMR SEI to a Ppicture immediately after the reference B picture in decoding order. Inparticular, a use of the reference B picture is a significantcharacteristic of the MPEG-4 AVC, in the random access basis having astructure such as I B Br B P B Br B P B Br B P . . . , a quadruple-speedplayback by decoding I and P pictures and a double-speed playback bydecoding I, P and Br pictures can be easily realized so that functionalcapability of trick-play is increased. In such a case, it is veryeffective that memory management without a breakdown can be guaranteed.Here, when an I picture is in a position other than the beginning of therandom access basis, the memory management command may be repeated in anI picture immediately after the I picture in decoding order using DRPMRSEI.

Here, the memory management command issued to the Br may be repeated ina picture which is different from a P picture or I picture immediatelyafter the picture in decoding order if it is guaranteed that there is apicture referred by the picture is in the DPB when a picture is to bedecoded at the IP playback. For example, in the case where the memorymanagement is not broken down even without being repeated in the Ppicture immediately after the picture in decoding order, it may betransmitted to a P picture which follows the P picture. Also, it may beguaranteed that the memory management is not broken down when only thereference I pictures or P pictures are decoded.

Further, the memory management command may be stored in a coded streamby the information other than the DRPMR or may be separately indicatedsuch as in database information.

Furthermore, it can be guaranteed that the memory management is notbroken down also when trick-play other than IP playback is performed.Hereafter, it is explained about an example when jump-in playback isperformed. A jump-in playback is an operation of displaying picturesstarting from a picture at a specified time. When starting displayingpictures from a picture other than a leading picture in the randomaccess unit, a picture necessary for decoding a picture to be displayedis decoded sequentially from a leading picture in the random accessunit. Here, in MPEG-4 AVC, a reference relationship is flexible.Therefore, the decoding processing for the jump-in playback or reverseplayback can be reduced by using a P picture in which a specificconstraint is given for decoding or referring to a picture (hereafterreferred to as Access Point (AP)-Picture). The AP-P picture hasfollowing two characteristics:

1. The AP-P picture can be decoded by selectively decoding I pictures orP pictures before the AP-P picture in the decoding order;

2. A picture after the AP-P picture in the decoding order does not referto a picture before the AP-P picture in the decoding order.

FIG. 12A is a drawing showing pictures and a memory management commandwhen the AP-P picture is used in the conventional technology. In thedrawing, a picture shown as AP-P indicates an AP-P picture. In order todecode AP-P25, it is only needed to decode I1, P7 and P16 so that P4,P10, P13 and P22 may be skipped. Thus, by selectively decoding pictures,the number of pictures necessary for decoding the AP-P picture placed atsome midpoint in the random access unit can be reduced. Consequently,the decoding processing when the playback is performed in the midpointin the random access unit can be reduced. Also, a picture after theAP-P25 in the decoding order does not refer to a picture before theAP-P25 in the decoding order. Further, a P picture necessary to bedecoded for decoding the AP-P picture can be indicated in a coded streamusing a SEI message and the like or in database information. Here, if amemory management command MMCO1 which instructs to transfer I1 to a longterm memory is stored in P4, the memory management command cannot beobtained when only a picture necessary for decoding the AP-P25 isdecoded.

FIG. 12B is a drawing showing pictures and a memory management commandwhen the AP-P picture is used in the first embodiment. As shown in FIG.12B, by repeating a memory management command MMCO1 in P7 which isdefinitely decoded when the AP-P 25 is decoded, it is found that it isnecessary to store I1 in a long-term memory when P7 is decoded. Thus, inthe case where a memory management command is issued to a picture whichis to be skipped when the AP-P picture is decoded, a memory managementwithout a breakdown can be realized by repeating the memory managementcommand in a P picture necessary for decoding the AP-P picture. Notethat, if it can be guaranteed that the memory management is not brokendown, the command may be repeated in the P picture that is notimmediately after the P picture with the original memory managementcommand but necessary for decoding the AP-P picture.

Further, more in general, when a picture necessary to be decoded fordecoding a specific P picture is displayed, it may be guaranteed thatmemory management can be realized by decoding only pictures necessary tobe decoded.

FIG. 13 is a flowchart of a coding method for generating a coded streamin which it is guaranteed memory management that is not broken down whenthe IP playback is performed. The processing from step S1001 to stepS1008 shows a processing for coding one picture which constitutes arandom access unit. First, in step S1001, it is judged whether a pictureto be coded is an I picture or a P picture. If it is either the Ipicture or the P picture, the processing moves on to step S1002, and ifit is not, the processing moves on to step S1004. In step S1002, it isjudged whether a memory management command is issued to a reference Bpicture which follows a P picture or an I picture immediately before thepicture to be coded in decoding order. In the case where the memorymanagement command is issued, the processing moves on to step S1003, andmoves on to step S1004 if the command is not issued. Here, in the casewhere there is no reference B picture immediately before the picture tobe coded in decoding order in the random access unit such as a leadingpicture in the random access unit, it is judged that the command is notissued. Following that, in step S1003, a DRPMR SEI in which the memorymanagement command is stored is generated. In the case where memorymanagement commands are issued to a plurality of reference B pictures,the contents of all memory management commands are included in the DRPMRSEI. Next, in step S1004, the picture data is coded and the processingmoves on to step S1005. In step S1005, it is judged whether or not amemory management command is issued to the current picture. If thecommand is issued, the processing moves on to step S1006, and moves onto step S1008 if the command is not issued. In step S1006, it is judgedwhether or not the current picture is the reference B picture. If thepicture is the reference B picture, the processing moves on to stepS1007, and moves on to step S1008 if it is not. In step S1007, thecontents of the memory management command and information for specifyinga picture to which the memory management command is issued are stored.Lastly, in step S1008, the coded data is outputted. Here, in the casewhere the DRPMR SEI is generated in step S1003, the output coded dataincludes DRPMR SEI. Note that, in the case where a picture type is notdetermined at the step S1001, the processing from the step S1001 to stepS1003 may be performed after the step S1004. Further, the coded data ofa picture may be outputted on a picture-by-picture basis, or may beoutputted sequentially as the coding is completed.

FIG. 14 is a flowchart of a coding method for generating a coded streamin which memory management, that is not broken down when the AP-P isdecoded, is guaranteed. While the fundamental processing is same as theprocessing for the IP playback shown in FIG. 13, it differs in thejudgement processing in the steps S1101, S1102 and S1003. In the stepS1101, it is judged whether or not the current picture is an I pictureor a P picture necessary for decoding the AP-P picture. Next, in thestep S1102, it is judged whether a memory management command is issuedto a P picture or an I picture that follow the P picture or I picturenecessary for decoding an AP-P picture immediately before the currentpicture in decoding order and is unnecessary for decoding the AP-Ppicture, within the random access unit. Also, in the step S1103, it isjudged whether or not the current picture is the I picture or P pictureunnecessary for decoding the AP-P picture.

Here, in the case where the AP-P picture can be decoded by selectivelydecoding only the P picture prior to the AP-P picture, only the Ppicture may be judged whether it is necessary for decoding the AP-Ppicture. However, in the case where it is necessary to decode an Ipicture that is a head of a random access unit, it may be indicated thatdecoding is necessary to be performed on the I picture.

Furthermore, the present method can be applied not only limited to theAP-P pictures but also to pictures in general on which a specificconstraint is given to the predictive structure and the like.

Note that, by combining the processing shown in FIG. 13 and FIG. 14,memory management that is not broken down when the IP playback isperformed and when the AP-P decoding is performed can be realized. Forexample, an operation such as decoding effectively a picture to bejumped-in using the AP-P and starting the IP playback from there can berealized.

Further, in the case where a picture that is needed to be decoded whentrick-play is performed is indicated in supplemental information and thelike, a memory management command may be repeated so as to obtain amemory management command necessary for decoding by decoding onlypictures necessary for the decoding.

FIG. 15 is a block diagram showing a structure of a decoding apparatus2000 in the present embodiment. The decoding apparatus 2000 includes apicture type obtaining unit 2001, a decoding judgement unit 2002, amanagement command analyzing unit 2003, a DPB 2004 and a decoding unit2005. First, coded data Vin is inputted to the picture type obtainingunit 2001. The picture type obtainment unit 2001 obtains a picture typeof a picture by detecting a picture boundary from the coded data Vin,and inputs the picture type Ptd into the decoding judgement unit 2002.The decoding judgement unit 2002 judges, based on the picture type Ptd,whether or not to decode the picture, and inputs the judgement resultRep into the management command analyzing unit 2003 and the decodingunit 2005. The management command analyzing unit 2003 executes memorymanagement processing if a memory management command is repeated inpicture data when it is instructed to decode the picture based on thejudgement result Rep, by analyzing the repeated memory command(repetition information) and transmitting a management command Cmd tothe DPB. The decoding unit 2005, when it is instructed to decode thepicture based on the judgement result Rep, obtains the reference dataRef by issuing a request Req to obtain reference picture data to theDPB, decodes picture data PicDat obtained by the picture obtainmentunit, and outputs the decoded picture Vout. Note that, an originalmemory command included in slice data of a picture shall be executed bya unit that is not shown in the diagram.

Note that, the supplemental information for specifying a picture neededto be decoded when trick-play is performed may be stored in a codedstream such as a leading AU of the random access unit or in databaseinformation. Here, the decoding judgement unit 2002 may determine an AUto be decoded by analyzing the supplemental information.

FIG. 16 is a flowchart showing an operation of decoding a coded streamin which a memory management that is not broken down when IP playback isperformed in the decoding apparatus 2000 is guaranteed. First, in stepS2001, it is judged whether a picture to be decoded is an I picture or aP picture. When it is judged that the picture is either I picture or Ppicture, the operation moves on to step S2002. If the picture is otherthan the above, the processing of the picture is ended without decodingthe picture and performs processing on the next picture. In step S2002,it is judged whether or not the current picture includes the DRPMR SEI,if it includes the DRPMR SEI, the operation moves on to S2003, and if itdoes not, the operation moves on to step S2004. In step S2003, thememory management processing is executed by analyzing the contents ofthe DRPMR SEI and the operation moves on to step S2004. In the stepS2004, the picture is decoded. Note that, in the step S2003, memorymanagement processing is not performed if it has already done by thepreceding command that is in the slice header or in the DRPMR SEI.

FIG. 17 is a flow chart showing an operation when an AP-P picture isdecoded in a coded stream in which a memory management that is notbroken down when the AP-P picture is decoded is guaranteed. While thefundamental processing is same as the processing when the IP playback isperformed as shown in FIG. 16, it differs in judgement processing instep S2101. In step S2101, it is judged whether or not a picture to bedecoded is a picture necessary for decoding the AP-P picture. If thepicture is necessary for decoding the AP-P picture, the operation moveson to step S2002, and if it is not necessary, the processing on thepicture is ended and the processing on the next picture is performed.

When the memory management command is repeated using a method other thanthe DRPMR SEI, a memory management command is obtained by apredetermined method.

Note that, by combining the processing shown in FIG. 16 and FIG. 17, amemory management that is not broken down when the IP playback isperformed and when the AP-P playback is performed can be realized.

Here, in operations such as decoding only the I pictures and P pictureswhen the IP playback is performed, or skipping P pictures or I picturesunnecessary for decoding the AP-P picture when the AP-P picture isdecoded, flag information which guarantees that a memory managementcommand necessary for managing the DPB can be obtained from the pictureto be decoded may be set to the database information or coded stream andthe like. For example, in a Network Abstraction Layer (NAL) unit of aslice of the reference B picture, a value of a field called nal_ref_idcwhich indicates whether or not the slice is a slice of the referencepicture is set to a value of one or more. In the non-reference Bpicture, the same field is set to 0. Accordingly the nal_ref_idc fieldmay be flag information. Also, in the database information, a codec typeinformation that shows MPEG-4 AVC and MPEG-2 Video and so on may be usedas a flag.

Note that, in the above, it is explained about the MPG-4 AVC. However, asimilar method can be applied to other coding methods.

Second Embodiment

FIG. 18 is a block diagram showing a multiplexer in the presentembodiment.

A multiplexer 35 receives inputted video data, codes the inputted datainto a stream of the MPEG-4 AVC, multiplexes and records the followinginformation together with the stream: access information to AUs whichconstitutes the stream; and database information including supplementalinformation for determining an operation when trick-play is performed.The multiplexer 35 includes a stream attribute determination unit 11, acoding unit 12, a database information generation unit 32, amultiplexing unit 34, and a recording unit 16. Same marks are assignedto units which perform same operations as in the conventionalmultiplexer shown in FIG. 5, and the explanations about the same unitsare omitted in here. Note that, the coding method is not only limited tothe MPEG-4 AVC and other methods such as MPEG-2 Video and MPEG-4 Videomay be applied. Further, it may include a coding unit 1000 instead ofthe coding unit 12.

The stream attribute determination unit 11 determines a coding parameterfor coding the MPEG-4 AVC and a constraint matter relating to thetrick-play, and outputs these to the coding unit 12 and a playbacksupport information generation unit 33 as attribute information TYPE.Here, the constraint matter relating to the trick-play includesinformation about whether or not to apply a constraint for constitutinga random access unit in a stream of the MPEG-4 AVC stream, whether ornot include information indicating AUs to be decoded or displayed whenvariable speed playback and reverse playback are performed, or whetheror not to constrain predictive structure among AUs. The playback supportinformation generation unit 33 generates, based on the inputtedattribute information TYPE, support information HLP indicating whetheror not to have a random access structure, and outputs the generatedinformation to the multiplexing unit 34. The multiplexing unit 34generates a multiplexed data by multiplexing the coded data inputtedfrom the coding unit 12, database information INFO and supportinformation HLP, and outputs the multiplexed data to the recording unit16. Note that, the coding unit 12 may output the stream of the MPEG-4AVC by packetizing into a MPEG-2 Transport Stream (TS), a Program Stream(PS) and the like. Or, it may packet the stream using a method definedby applications such as BD.

FIG. 19A and FIG. 19B show an example of information indicated in thesupport information HLP. The support information HLP has following twomethods: a method of directly indicating information about a stream asshown in FIG. 19A; and a method of indicating whether or not the streamsatisfies a constraint defined by a specific application standard asshown in FIG. 19B. In FIG. 19A, the followings are indicated asinformation of the stream: i) whether the stream has a random accessstructure; ii) whether there is a constraint on a predictive structureamong pictures stored in an AU; and iii) whether there is informationindicating an AU which is decoded or an AU which is displayed whentrick-play is performed.

Here, the information of the AU which is decoded or displayed when thetrick-play is performed may directly indicate the AU which is decoded ordisplayed, or may indicate priority of decoding or displaying. Forexample, information indicating AU which is decoded or displayed on arandom access unit-by unit basis can indicate that it is stored in a NALunit which has a special type defined by the application. Here, it mayindicate whether there is information indicating a predictive structureamong AUs which constitutes a random access unit. Further, informationabout AU to be decoded or displayed when trick-play is performed may beadded together for each group of more than one random access units ormay be added to each AU which constitutes a random access unit.

Furthermore, when the information indicating an AU to be decoded ordisplayed is stored in the NAL unit which has a special type, it mayindicate a NAL unit type of the NAL unit. In an example of FIG. 20, inthe support information HLP, information about an AU to be decoded ordisplayed when trick-play is performed is included in a NAL unit whoseNAL unit type is 0. Herein, by separating the NAL unit whose NAL unittype is 0 from the AU data in the stream, information relating to thetrick-play can be obtained.

Further, the constraint of the predictive structure may indicate whetheror not to satisfy one or more predetermined constraint matters orwhether or not to satisfy the following individual constraint.

i) The respective AUs of I picture and P picture have same decoding anddisplaying orders.

ii) The AU of the P picture does not refer to the AU of the B picture.

iii) An AU displayed after a leading AU in the random access unit onlyrefers to AUs included in the random access unit.

iv) Each AU can only refer to the maxim N numbers of AUs in decodingorder. Herein, an AU is counted for each reference AU or all AUs and thesupport information HLP may indicate the value of N.

Note that, in MPEG-4 AVC, in order to improve picture quality, a pictureon which filtering (deblocking) for reducing block distortion isperformed after decoding the picture is used as a reference picture,while a picture before the deblocking can be used as a picture fordisplay. Herein, it is necessary for the picture decoding apparatus tostore picture data before and after deblocking. Here, informationindicating whether or not it is necessary to store the picture beforedeblocking as a picture for display may be stored in support informationHLP.

Here, the support information HLP may include all of the aboveinformation or may include a part of the information. In addition, itmay include necessary information based on a predetermined condition.For example, it may include information about whether there istrick-play information only in the case where there is no constraint ona predictive structure.

Further, the support information HLP may include information indicatingthe following: whether or not the IP playback can be realized withoutcausing breakdown of the memory management by decoding only the Ipictures and the P pictures; or whether or not the AP-P picture can bedecoded without causing breakdown of the memory management by decodingonly the I pictures or P pictures necessary for decoding the AP-Ppicture.

Further, information other than the above may be included in the supportinformation HLP.

In FIG. 19B, the support information HLP shows not directly informationrelating to a structure of a stream but whether or not satisfyconstraints relating to the stream structure defined by a HD DVDstandard that is a standard for storing high precision picture of HighDefinition (HD) into a DVD. Furthermore, in an application standard suchas BD-ROM, in the case where a plurality of modes are defined for theconstraint on the stream structure, information indicating which mode isapplied may be stored in the support information HLP. For example, itcan be used that a mode 1 does not have constraints, and a mode 2 has arandom access structure and information for specifying an AU which isdecoded when trick-play is performed is included in a stream. Here, itmay indicate whether or not to satisfy constraints defined in acommunication service such as downloading and streaming, or broadcaststandard.

Note that, the support information HLP may indicate both informationshown in FIG. 19A and FIG. 19B. Also, in the case where it has beenknown that a stream satisfies a constraint in a particular applicationstandard, it may not indicate about whether the stream satisfies anapplication standard, but store the constraint on the applicationstandard by converting into a method of directly describing a streamstructure as shown in FIG. 19A.

Here, in the case where information indicated in the support informationHLP changes during the streaming, information of each section may berespectively stored. For example, in the case where different streamsare edited and connected to each other, in the edited stream, thesupport information HLP may change during the streamlining. Therefore,the contents of the support information HLP are also switched.

Note that, the information indicating an AU to be decoded or displayedwhen trick-play is performed may be stored as database information.

FIG. 21 is a flowchart showing an operation of the multiplexer 35. In astep s11, attribute information TYPE is determined based on a usersetting or a predetermined condition. In a step s12, a stream is codedbased on the attribute information TYPE. In a step s13, supportinformation HLP is generated based on the attribute information TYPE.Following that, in a step s14, access information is generated for eachaccess basis of the coded stream, and generates database informationINFO together with other necessary information. In a step s15, thesupport information HLP and the database information INFO aremultiplied, and the multiplexed data is recorded in a step s16. Here,the operation of the step s13 may be performed before the operation ofstep s12, or it may be performed after the operation of step s14.

Third Embodiment

FIG. 22 is a block diagram showing a structure of a second multiplexerin the present embodiment.

A multiplexer 43 receives a packetized stream which is distributed froma server that is not shown in the diagram, multiplexes and records,together with the stream, general database information including accessinformation to AUs that make up a stream and supplemental informationfor determining an operation when trick-play is performed. It includes astream attribute obtainment unit 41, a stream receiving unit 42, adatabase information generation unit 32, a multiplexing unit 34, and arecording unit 16. Same marks are attached to units which perform sameoperations as units in the multiplexer explained in the secondembodiment, and the explanations about same units are omitted in here.

The stream attribute obtainment unit 41 generates attribute informationTYPE based on stream information obtained separately from the stream,and outputs the attribute information TYPE to the playback supportinformation generation unit 33. Here, the stream information includesinformation relating to trick-play such as: whether or not to apply aconstraint for constituting a random access unit in a stream of theMPEG-4 AVC; whether or not to include information indicating AUs to bedecoded or displayed when variable speed playback and reverse playbackare performed; and whether or not to give a constraint on a predictivestructure between AUs. The stream receiving unit receives a stream ofthe MPEG-4AVC, that is packetized by a MPEG-2 Transport Stream (TS) anda Real-time Transmission Protocol (RTP), outputs the received stream tothe multiplexer 34 as a stream for recording, and also outputs theaccess information to the general database information generation unit14.

Here, when the TS packet, RTP packet and the like are received in anenvironment where packet loss is generated, in the case where errorconcealment processing is performed when information and data indicatingthat data in a stream is lost because of the packet loss, the HLP maystore information about that as support information. As informationindicating the loss of data, the following information can be shown:flag information indicating whether or not data in a stream is lost;information indicating to insert a special error notification code intoa stream in order to notify the lost part; or identification informationof an error notification code to be inserted.

Fourth Embodiment

FIG. 22 is a block diagram showing a structure of a demultiplexer in thepresent embodiment.

A demultiplexer 55 separates a MPEG-4 AVC stream from the multiplexeddata generated by the multiplexers explained in the second and thirdembodiments, and plays back the separated MPEG-4 AVC. It includes adatabase information analyzing unit 51, a trick-play operationdetermination unit 53, a decoding/displaying AU determination unit 54,an AU separation unit 24, a decoding unit 25, and a displaying unit 26.Here, same marks are attached to the units which perform same operationsas the units in the conventional demultiplexer shown in FIG. 6 and theexplanations about the same units are omitted.

The database information analyzing unit 51 includes a playback supportinformation analyzing unit 52 and a general database informationanalyzing unit 22. The playback support information analyzing unit 52,when a trick-play instruction signal is inputted, obtains and analyzessupport information HLP from the database information in the multiplexeddata, generates trick-play support information based on the analysisresult, and notifies the trick-play operation determination unit 53 ofthe trick-play support information. The trick-play operationdetermination unit 53 determines a method of determining an AU to bedecoded and displayed when the trick-play is performed based on thetrick-play support information, and notifies the decoding/displaying AUdetermination unit 54 of a trick-play mode indicating the determinedmethod. The decoding/displaying AU determination unit 54 analyzes thetrick-play information TRK obtained by the AU separation unit 24,determines an AU to be decoded and displayed by a method indicated bythe trick-play mode MODE, and notifies the AU separation unit 24 and thedisplaying unit respectively of identification information of the AU tobe decoded and identification information of the AU to be displayed.Here, the AU to be displayed may be determined by thedecoding/displaying AU determination unit 54 based on a specifiedplayback speed and the like. Further, when the trick-play informationTRK is stored in the database information, the trick-play informationTRK stored in the database information may be obtained by settinganother unit in the database information analyzing unit 51.

FIG. 24 is a flowchart showing operations of the demultiplexer 55. Whena trick-play instruction signal is inputted, it obtains supportinformation HLP from the multiplexed data in step s20. In a step s21, anoperation of determining an AU to be decoded and displayed based on theobtained support information HLP. In a step s22, it is judged whether ornot it is determined to use the trick-play information TRK when thetrick-play is performed. In a step s23, the trick-play information TRKis obtained from the stream and analyzed, and the operation moves on toa step s24. If the trick-play information TRK is not used, the operationmoves on directly to the step s24. In the step s24, an AU to be decodedand displayed is determined based on the method determined in the steps21, and the operation moves on to a step s25. In the step s25, thedetermined AU is decoded and displayed. Note that, the supportinformation HLP may be obtained only in the case where the trick-play isperformed when the playback is started or after the playback is started.

FIG. 25 is a flowchart showing contents of the processing in the steps21. Hereafter, judging in steps s30, s33, and s35 are performed basedon the trick-play support information obtained from the supportinformation HLP. In the step s30, it is judged whether or not the streamhas a random access structure, the operation moves on to the step s31 ifthe stream has a random access structure, and the operation moves on tothe step s32 if the stream does not have the random access structure.Then, respective AU to be decoded is determined. In the step s31, it isdetermined to start decoding from a leading AU in the random accessunit. In the step s32, it is determined to start decoding from the AUwhen the leading AU in the random access unit is an AU of an IDRpicture, the decoding is started from the AU. Here, in the case where adisplay time of an AU including the preceding IDR picture is before apredetermined time or more, an AU to be coded first may be determinedbased on a predetermined rule such as starting decoding of an leading AUin N preceding access basis or an I picture other than the IDR. In thestep s33, it is judged whether or not the trick-play information TRK isincluded in the stream. If the TRK is included, the operation moves onto a step s34, and if the TRK is not included, the operation moves on toa step s35. In the step s34, the processing is ended by determining anAU to be decoded or displayed based on the trick-play information TRK.In the step s35, it is judged whether or not there is a constraint onthe predictive structure between AUs. If there is a constraint, theoperation moves on to a step s36, and if there is no constraint, theoperation moves on to a step s37. In the step s36, the processing isended by determining that only an AU which needs to be decoded whendecoding the AU which is necessary to be displayed when variable speedplayback and reverse playback are performed based on the constraint onthe predictive structure. Also, in the step s37, the processing is endedby determining that all AUs are decoded. Consequently, a method ofdetermining an AU to be decoded first is determined as the results ofthe steps s31 and s32, and a method of specifying an AU to be decodedwhen the variable speed playback or trick-play is performed as theresults of the steps s34, s36 and s37. Then, they are outputted to thedecoding/displaying AU determination unit 54 as information of therespective trick-play modes MODE. Note that, when a jump-in playback isperformed, the processing may be ended after the step 32 or the steps31. Here, as a method of determining an AU to be decoded when thetrick-play is performed, a predetermined method may be used: forexample, in the case where it is judged that the trick-play informationTRK is not included in the stream in the step s33, only AUs of Ipictures and P pictures are decoded, or only AUs of I pictures, Ppictures, and B pictures to be referred are decoded.

Note that, in the case where information for determining an AU to bedisplayed is included in the trick-play information TRK, informationindicating to determine an AU to be displayed based on the trick-playinformation TRK may be included in the trick-play mode MODE.

Here, in the case where decoding by a method determined by thetrick-play operation determination unit 53 cannot be realized, an AU tobe decoded by a predetermined method may be determined. For example, inthe case where it is indicated to obtain the playback information TRKfrom the coded stream by the trick-play mode MODE, if the trick-playinformation TRK cannot be obtained in the coded stream, all AUs aredecoded or AUs to be decoded can be determined based on otherinformation obtained from the support information HLP. Herein, it may beverified about whether the database information includes the trick-playinformation TRK.

Here, in the case where information other than the trick-play relatedinformation is included in the support information HLP, a decoding ordisplaying operation may be switched according to the information. Forexample, the operations may be switched based on packet loss informationwhen data received through broadcast and communication is recorded.

Further, a medium on which the multiplexed data is recorded is not onlylimited to an optical disc but it may be other recording mediums such asa hard disc and a nonvolatile memory.

Furthermore, operations of the decoding/displaying AU determination unit23 differ from each other. By preparing conventional demultiplexersshown in FIG. 6, a demultiplexer to be used may be switched based on atrick-play mode determined by separately set playback supportinformation analyzing unit 52 and trick-play operation determinationunit 53. For example, any two out of the conventional demultiplexershaving the decoding/displaying AU determination units 23 which performthe following three types of operations are prepared, and may switch thedemultiplexer to be used based on the support information HLP of themultiplexed data to be played back. The three types are: i) to determinea demultiplexer so as to decode all AUs all the time; ii) to obtain thetrick-play information TRK all the time and determine an AU to bedecoded; and iii) to determine an AU to be decoded by presuming that astream follows a specific predictive structure.

Fifth Embodiment

As a method of recording multiplexed data onto an optical disc by amultiplexer according to the second embodiment, it is explained about amethod of storing support information HLP as database information of aBD that is a next generation optical disc.

First, it is explained about a recording format of a BD-ROM.

FIG. 26 is a diagram showing a structure of a BD-ROM, in particular astructure of a BD disc (104) that is a disc medium and data (101, 102,and 103) recorded on the disc. The data to be recorded on the BD disc(104) are AV data (103), BD database information (102) such as databaseinformation relating to the AV data and AU playback sequence, and a BDplayback program (101) for realizing an interactive. In the presentembodiment, it is explained about a BD disc mainly for an AV applicationfor playing back AV contents in a movie for the purpose of explanation.However, there is no doubt that it is same even if it is used for otheruses.

FIG. 27 is a diagram showing a directory/file structure of logical datarecorded on the BD disc. The BD disc has a recording area in a spiralform from the inner radium toward the outer radius as similar to, forexample, DVD, CD and the like, and has a logical address space in whichlogical data can be recorded between a lead-in of the inner radium and alead-out of the outer radium. Also, there is a special area inside thelead-in which is read only by a drive called a Burst Cutting Area (BCA).This area cannot be read from the application so that it may be used,for example, for a copyright protection technology and the like.

In the logical address space, application data such as video data leadby the final system information (volume) is recorded. As explained inthe conventional technology, the file system is a UDF, an ISO96660 andthe like. It allows reading out logical data stored as in the ordinalpersonal computer PC, using a directory and a file structure.

In the present embodiment, as the directory and file structure on the BDdisc, a BDVIDEO directory is placed immediately under a root directory(ROOT). In this directory, data (101, 102 and 103 explained in FIG. 26)such as AV contents and database information dealt in the BD are stored.

Under the BDVIDEO directory, the following seven types of files arerecorded:

BD. INFO (file name fixed)

A file that is one of “BD database information” and informationconcerning the BD disc as a whole is recoded in the file. A BD playerfirstly reads out this file.

BD. PROG (file name fixed)

A file that is one of “BD playback program” and playback controlinformation concerning the BD disc as a whole is recorded in the file.

XXX. PL (“XXX” is variable, an extension “PL” is fixed)

A file that is one of “BD database information” and play listinformation that is a scenario (playback sequence) is recorded in thefile. There is one file for each play list.

XXX. PROG (“XXX” is variable, an extension “PROG” is fixed)

A file that is one of “BD playback program” and playback controlinformation for each play list is recorded in the file. A correspondencewith a play list is identified by a file body name (“XXX” matches).

YYY. VOB (“YYY” is variable, an extension “VOB” is fixed)

A file that is one of “AV data” and VOB (same as VOB explained in theconventional example) is recorded in the file. There is one file foreach VOB.

YYY. VOBI (“YYY” is variable, an extension “VOBI” is fixed)

A file that is one of “BD database information” and stream databaseinformation concerning VOB that is AV data is recorded in the file. Acorrespondence with a VOB is identified by a file body name (“YYY”matches).

ZZZ. PNG (“ZZZ” is variable, an extension “PNG” is fixed)

A file that is one of “AV data” and image data PNG (a picture formatstandardized by W3C, and pronounced as “ping”) for structuring subtitlesand a menu in the file. There is one file for each PNG image.

With references to FIGS. 28 to 32, it is explained about a structure ofnavigation data of BD (BD database information).

FIG. 28 is a diagram showing an internal structure of a VOB databaseinformation file (“YYY. VOBI”).

The VOB database information has stream attribute information(Attribute) of the VOB and a time map (TMAP). There is a streamattribute for each of video attribute (Video) and audio attribute(Audio#01 to Audio#m). In particular, in the case of the audio stream,since the VOB can have audio streams at the same time, the number ofaudio streams (Number) indicates whether there is a data field or not.

The followings indicate fields of video attribute (Video) and valuesthereof.

Compression method (Coding):

-   -   MPEG1    -   MPEG2    -   MPEG3    -   MPEG4 (Advanced Video Coding)

Resolution:

-   -   1920×1080    -   1440×1080    -   1280×720    -   720×480    -   720×565

Aspect Rate

-   -   4:3    -   16:9

Framerate

-   -   60    -   59.94 (60/1.001)    -   50    -   30    -   29.97 (30/1.001)    -   24    -   23.976 (24/1.001)

The followings are fields of the audio attribute (Audio) and valuesthereof.

Compression method (Coding):

-   -   AC3    -   MPEG1    -   MPEG2    -   LPCM    -   The number of channels (Ch):    -   1 to 8    -   Language Attribute (Language)

A time map (TMAP) is a table having information for each VOBU. The tableincludes the number of VOBU (Number) held by the VOB and each VOBUinformation (VOBU#1 to VOBU#n). Each of the VOBU information is made upof an address I_start of an address of a VOBU leading TS packet (Ipicture start), an offset address (I#end) until the end address of the Ipicture, and a playback start time (PTS) of the I picture. In the casewhere a stream of the MPEG-4 AVC has a random access structure, a VOBUcorresponds to one or more random access units.

FIG. 29 is a diagram explaining details of the VOBU information.

As widely known, there is a case that MPEG video stream is compressedinto variable bit rate in order to recording in high picture quality,and there is no simple correspondence between the playback time and thedata size. In contrary, an AC3 that is a compression standard of audiocompresses audio in a fixed bit rate. Therefore, a relationship betweena time and an address can be obtained by a primary expression. However,in the case of a MPEG video data, each frame has a fixed display time,for example, in the case of NTSC, one frame has a display time of1/29.97 seconds. However, data size of each compressed frame largelychanges depending on a characteristic of a picture and a picture typeused for compression, specifically I/P/B pictures. Therefore, in thecase of MPEG video, a relationship between time and address cannot bedescribed in a primary expression.

As a matter of fact, it is impossible to describe a MPEG system streamwhich is obtained by multiplexing the MPEG video data in a form of aprimary expression. Specifically, VOB also cannot describe time and datasize in a primary expression. Therefore, a time map (TMAP) is used forconnecting a relationship between time and address in the VOB.

Therefore, when time information is given, first, it is searched inwhich VOBU the time belongs (tracks PTS for each VOBU), skips the PTSimmediately before the time to a VOBU having a TMAP (address specifiedby I#start), starts decoding pictures from a leading I picture in theVOBU, and starts displaying pictures from a picture of the time.

Next, with reference to FIG. 30, it is explained about an internalstructure of play list information (“XXX. PL”).

The play list information is made up of a cell list (CellList) and anevent list (EventList).

The cell list (CellList) is a playback sequence in a play list, and thecell is played back in an order of description on the list. The contentsof the cell list (CellList) include the number of cells (Number) andeach cell information (Cell#1 to Cell#n).

The cell information (Cell#) includes a VOB file name (VOBName), a starttime (In) and end time (Out) in the VOB, and a subtitle table. The starttime (In) and end time (Out) are described by frame numbers in each VOB,and an address of the VOB data necessary for playback can be obtained byusing the time map (TMAP).

The subtitle table is a table having subtitle information to be playedback at the same time with the VOB. The subtitle can have a plurality oflanguages similar to audio, and first information of the subtitle tableis made up of a number of languages (Number) and the following table foreach language (Language#1 to Language#k).

Each language table (Language#) is made up of a language information(Lang), the number of subtitle information (Number) which is separatelydisplayed, and subtitle information of the subtitle (Speech#1 toSpeech#j). The subtitle information (Speech#) is made up of acorresponding image data file name (Name), a subtitle display start time(In), subtitle display end time (Out), and a display position of thesubtitle (Position).

The event list (EventList) is a table in which events generated in theplay list are defined. The event list is made up of each event (Event#1to Event#m) following to the number of events (Number). Each event(Event#) is made up of a type of event (Type), an event ID (ID), anevent generation time (Time) and a duration.

FIG. 31 is an event handier table (“XXX. PROG”) having an event handier(time event and user event for menu selection) for each play list.

The event hander table has a defined event handler/the number ofprograms (Number) and individual event handler/program (Program#1 toProgram#n). The description in each event handler/program (Program#) hasa definition about event handler start (<event_handler>tag) an ID of anevent handler paired with the event. After that, the program isdescribed between curly brackets “{” and “}I” following to a Function.The events (Event#1 to Event#m) stored in an event list of the “XXX. PL”is specified using an ID of an event handler.

Next, with reference to FIG. 32, it is explained about an internalstructure of information concerning a BD disc as a whole (“BD. INFO”).

The BD disc total information is made up of a title list and an eventtable for a global event.

The title list is made up of a number of titles in a disc (Number) andthe following each title information (Title#1 to Title#n). Each titleinformation (Title#) includes a play list table (PLTable) included in atitle and a chapter list in the title. The play list table (PLTable)includes the number of play lists in the title (Number) and a play listname (Name), specifically, a file name of the play list.

The chapter list is made up of a number of chapters (Number) included inthe title and individual chapter information (Chapter#1 to Chapter#n).Each pieces of the chapter information (Chapter#) has a table of cellsincluded in the chapter. The cell table is made up of a number of cells(Number) and individual cell entry information (CellEntry#1 toCellEntry#k). The cell entry information (CellEntry#) is described witha play list name including the cell and a cell number in the play list.

The event list (EventList) has a number of global events (Number) andindividual global event information. Here, it should be mentioned that aglobal event defined first is called a first event, which is an event tobe called first when a BD disc is inserted to a player. The eventinformation for global event only has an event type (Type) and event ID(ID).

FIG. 34 shows a table of a program of a global event handler (“BD.PROG”). This table is same as the event handler table explained in FIG.32.

In such BD-ROM format, the support information HLP is stored as streamattribute information of the VOB database information. When the supportinformation HLP is used only for the MPEG-4 AVC, the support informationHLP may be stored only when the compression method is the MPEG-4 AVC.

Note that, in addition to the stream attribute information and a timemap, the support information HLP may be stored by setting an area forstoring the playback support information in the VOB databaseinformation. Also, the support information HLP may be stored as BDdatabase information other than the VOB database information.

Further, the support information HLP may be stored not only in theBD-ROM format but also in other recording formats such as a BD-RE(Rewritable) as database information.

Sixth Embodiment

FIG. 34 is a block diagram roughly showing a functional structure of aplayer which plays back data recorded on the BD disc according to thefifth embodiment.

The data recorded on a BD disc (201) are read out through optical pickup(202). The read data is transferred to a special memory depending onrespective type of data. The BD playback program (contents of “BD. PROG”or “XXX. PROG” files), the BD database information (“BD. INFO”, “XXX.PL” or “YYY. VOBI”), and the AV data (“YYY. VOB” or “ZZZ. PNG”) arerespectively transferred to a program recording memory (203), a databaseinformation recording memory (204) and an AV recording memory (205).

The BD playback program recorded in the program recording memory (203),the BD database information recorded in the database informationrecording memory (204), and the AV data recorded in the AV recordingmemory (205) are respectively processed by a program processing unit(206), a database information processing unit (207), and a presentationprocessing unit (208).

The program processing unit (206) processes a program for receivinginformation about play lists to be played back by the databaseinformation processing unit (207) and event information such as timingof executing a program. Also, the program can dynamically change theplay lists to be played back. In this case, it can be realized bysending an instruction of playing back play list to the databaseinformation processing unit (207). The program processing unit (206)receives an event from a user, specifically, a request sent from aremote controller key, and executes the event if there is a programcorresponding to the user event.

The database information processing unit (207) receives an instructionof the program processing unit (206), analyzes the corresponding playlist and database information of a VOB corresponding to the play list,and instructs to play back target AV data to the presentation processingunit (208). Further, the database information processing unit (207)receives standard time information from the presentation processing unit(208), instructs the presentation processing unit (208) to stop the AVdata playback based on the time information, and further generates anevent indicating a program executing timing for the program processingunit (206).

The presentation processing unit (208) has decoders respectivelycorresponding to video, audio, and subtitle/image (still picture). Eachof the decoders decodes AV data based on an instruction sent from thedatabase information processing unit (207), and outputs the decoded AVdata. The video data, subtitle and image are respectively described on aspecial plane, a video plane (210) and an image plane (209) after theyare decoded, and synthesized the images by the synthesizing unit (211)and outputted to a display device such as a television.

Hereafter, it is explained about player operations when trick-play isperformed.

The database information processing unit 207 includes a function of thetrick-play operation determination unit 53 in the demultiplexer 55according to the fourth embodiment, when a trick-play instruction signalto perform trick-play such as variable speed playback, reverse playbackor jump-in playback is inputted via the program processing unit 206,obtains and analyzes the support information HLP from the databaseinformation memory 204, and determines a method of determining operationof decoding and displaying when trick-play is performed. Thepresentation processing unit 208 includes a function of thedecoding/displaying AU determination unit 54 in the demultiplexer 55,determines an AU to be decoded and displayed based on the methoddetermined by the database information processing unit 207, and decodedand displayed the determined AU. Here, the database informationprocessing unit 207 may have the function of the decoding/displaying AUdetermination unit 54.

Further, when the trick-play information TRK is stored in the BDdatabase information, the database information processing unit 207obtains the trick-play information TRK from the database informationmemory 204. The obtained trick-play information TRK is analyzed in thepresentation processing unit 208.

Note that each function block in the block diagram shown in FIGS. 10,15, 18, 22 and 23 can be realized as an LSI that is an integratedcircuit apparatus. Such LSI may be incorporated in one or plural chipform (e.g. function blocks other than a memory may be incorporated intoa single chip). Here, LSI is taken as an example, however, it may becalled “IC”, “system LSI”, “super LSI” and “ultra LSI” depending on theintegration degree.

The method for incorporation into an integrated circuit is not limitedto the LSI, and it may be realized with a private line or a generalprocessor. After manufacturing of LSI, a Field Programmable Gate Array(FPGA) that is programmable, or a reconfigurable processor that canreconfigure the connection and settings for the circuit cell in the LSI,may be utilized.

Furthermore, along with the arrival of technique for incorporation intoan integrated circuit, which replaces the LSI owing to a progress insemiconductor technology or another technique that has deviated from it,integration of the function blocks may be carried out using thenewly-arrived technology. Application of bio-technology may be cited asone of the examples.

Among the function blocks, only a unit for storing data may beconstructed separately without being incorporated in a chip form, as thestorage medium 115 described in the present embodiment.

Note that the main part in the function blocks shown in FIGS. 10, 15,18, 22 to 25 and 34 or in the flowcharts shown in FIGS. 13, 14, 16 and17 can be realized by a processor or a program.

As stated above, it is possible to employ the picture coding method andthe picture decoding method presented in the above embodiment in any oneof the above-described devices and systems. Accordingly, it becomespossible to achieve the effects described in the aforementionedembodiment.

Seventh Embodiment

In addition, by recording a program for realizing the layout of themoving picture coding method or the moving picture decoding method asshown in each of the above-mentioned embodiments, on a recording mediumsuch as a flexible disk, it becomes possible to perform the processingas shown in each of the above embodiments easily in an independentcomputer system.

FIG. 35A, FIG. 35B and FIG. 35C are diagrams of a recording medium forrecording a program for realizing the moving picture coding method andthe moving picture decoding method in the above embodiments in thecomputer system.

FIG. 35B shows the front view of a flexible disk and the schematiccross-section, as well as a flexible disk itself, whereas FIG. 35A showsan example of a physical format of the flexible disk as a recordingmedium itself. A flexible disk FD is contained in a case F, a pluralityof tracks Tr are formed concentrically on the surface of the disk in theradius direction from the periphery, and each track is separated into 16sectors Se in the angular direction. Therefore, in the flexible diskstoring the above-mentioned program, the above program are recorded inan area allocated for it on the above flexible disk FD

In addition, FIG. 35C shows the configuration for recording and playingback the program on and from the flexible disk FD. When the program isrecorded on the flexible disk FD, the computer system Cs writes in themoving picture coding method and the moving picture decoding method asthe program on the flexible disk FD via a flexible disk drive. When theabove moving picture coding method and the moving picture decodingmethod are constructed in the computer system using the program recordedon the flexible disk, the program is read out from the flexible disk viathe flexible disk drive and transferred to the computer system.

Note that the above explanation is made on an assumption that arecording medium is a flexible disk, but the same processing can also beperformed using an optical disk. In addition, the recording medium isnot limited to these, but any other mediums such as a CD-ROM, memorycard, and a ROM cassette can be used in the same manner if a program canbe recorded on them.

Although only some exemplary embodiments of this invention have beendescribed in detail above, those skilled in the art will readilyappreciate that many modifications are possible in the exemplaryembodiments without materially departing from the novel teachings andadvantages of this invention. Accordingly, all such modifications areintended to be included within the scope of this invention.

A multiplexer and a demultiplexer according to the present invention canperform efficient decoding or displaying when data obtained bymultiplexing a stream of MPEG-4 AVC is special-played back, so that thepresent invention is particularly effective for a playback device of apackage media which focuses on a trick-play function.

1. A stream generation apparatus that generates a stream including codedpictures, the coded pictures including a first unit of storage in whichfirst information obtained by coding a command for managing a bufferthat holds a decoded picture as a reference picture is stored, and asecond unit of storage which is located before the first unit of storageand for storing second information, and when a coded picture stores thesecond information obtained by coding content identical to the commandindicated by the first information of another coded picture which islocated before the coded picture in decoding order, the first unit ofstorage, the second unit of storage, and the command are definedaccording to the MPEG-4 AVC standard, stream generation apparatuscomprising: a judging unit configured to judge whether or not a firstcoded picture in which the first information is stored in the first unitof storage is a coded picture corresponding to a reference B picture tobe referable to when another picture is decoded; a coding unitconfigured to code, as the second information, content identical to thecommand indicated by the first information, when the judgment unitjudges that the first coded picture is a coded picture corresponding tothe reference B picture to be referable to when the other picture isdecoded; and a generating unit configured to generate a stream bystoring the second information in the second unit of storage in a secondcoded picture which corresponds to an I picture which is located afterthe first coded picture in decoding order, or a P picture.
 2. A streamgeneration method of generating a stream including coded pictures, thecoded pictures including a first unit of storage in which firstinformation obtained by coding a command for managing a buffer thatholds a decoded picture as a reference picture is stored, and a secondunit of storage which is located before the first unit of storage andfor storing second information, and when a coded picture stores thesecond information obtained by coding content identical to the commandindicated by the first information of another coded picture which islocated before the coded picture in decoding order, the first unit ofstorage, the second unit of storage, and the command are definedaccording to the MPEG-4 AVC standard, the stream generation methodcomprising: judging whether or not a first coded picture in which thefirst information is stored in the first unit of storage is a codedpicture corresponding to a reference B picture to be referable to whenanother picture is decoded; coding, as the second information, contentidentical to the command indicated by the first information, when thejudging judges that the first coded picture is a coded picturecorresponding to the reference B picture to be referable to when theother picture is decoded; and generating a stream by storing the secondinformation in the second unit of storage in a second coded picturewhich corresponds to an I picture which is located after the first codedpicture in decoding order, or a P picture.
 3. A non-transitorycomputer-readable storage medium on which a stream including codedpictures is recorded, the coded pictures including a first unit ofstorage in which first information obtained by coding a command formanaging a buffer that holds a decoded picture as a reference picture isstored, and a second unit of storage which is located before the firstunit of storage and for storing second information, and when a codedpicture stores the second information obtained by coding contentidentical to the command indicated by the first information of anothercoded picture which is located before the coded picture in decodingorder, in the case where the first information is stored in the firstunit of storage of a first coded picture and where the first codedpicture is a coded picture corresponding to a reference B picture to bereferable to when another picture is decoded, the second information isstored in the second unit of storage of I picture or P picture which islocated after the first coded picture in decoding order, the secondinformation being identical to the first information in the first codedpicture, and the first unit of storage, the second unit of storage, andthe command are defined according to the MPEG-4 AVC standard.